Tracking changes that affect performance of deployed applications

ABSTRACT

An application monitoring infrastructure that enables application configuration changes on multiple machines across multiple OS types to be tracked by identifying data containers that are to be monitored for changes, detecting a change to a monitored data container, and storing data representative of a changed version of the monitored data container responsive to detecting that the monitored container was changed. The data containers that are to be monitored for changes are identified from templates, and a unique template is provisioned for each of the applications.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.13/491,305, entitled “Tracking Changes That Affect Performance ofDeployed Applications,” filed Jun. 7, 2012, which is related to U.S.patent application Ser. No. 13/491,329, entitled “CorrelatingPerformance Degradation of Applications to Specific Changes Made toApplications,” filed Jun. 7, 2012.

BACKGROUND

With the transition to virtualized infrastructure in recent years,outages due to configuration changes to applications have been on therise. Some of the more common configuration changes include code push,and changes to the clustering configuration, number of threads allocatedto applications, and datastore configuration. Although many of thesechanges are tested prior to rolling them out in the productionenvironment, outages and significant performance degradations are notdiscovered, indeed cannot be discovered, if they result from scaleissues, e.g., when the changes are rolled out in a cloud environmenthaving virtual machines that number in the millions.

SUMMARY

One or more embodiments disclosed herein provide an applicationmonitoring infrastructure that enables application configuration changeson multiple machines across multiple OS types to be tracked, andcorrelated to performance degradation and outages.

A method for tracking configuration changes of applications, accordingto an embodiment, includes the steps of identifying data containers thatare to be monitored for changes, detecting a change to a monitored datacontainer, and storing data representative of a changed version of themonitored data container responsive to detecting that the monitoredcontainer was changed, wherein the data containers that are to bemonitored for changes are identified from templates, and a uniquetemplate is provisioned for each of the applications.

A method for tracking configuration changes of applications, accordingto another embodiment, includes the steps of identifying data containersthat are to be monitored for changes, receiving notifications of fileevents, transforming multiple file events into a normalized file event,detecting a change to a monitored data container based on the normalizedfile event, and storing data representative of a changed version of themonitored data container responsive to detecting that the monitoredcontainer was changed.

Further embodiments of the present invention include, withoutlimitation, a non-transitory computer-readable storage medium thatincludes instructions that enable a computer system to implement one ormore aspects of the above methods as well as a computer systemconfigured to implement one or more aspects of the above methods.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computing environment in which oneor more embodiments may be practiced.

FIG. 2 is a conceptual diagram that illustrates various softwarecomponents of a monitored machine that enable tracking of changes madeto applications running in the monitored machine.

FIG. 3 is a flow diagram of a method for generating change events when achange to a folder or a file is detected.

FIG. 4 is a flow diagram of a method for versioning changes made tofiles of monitored applications.

FIG. 5 is a flow diagram of a method for generating, and processinginputs made to, an application performance monitoring user interface.

FIG. 6A illustrates a sample application performance monitoring userinterface.

FIG. 6B illustrates a sample application change tracking user interface.

DETAILED DESCRIPTION

FIG. 1 is a simplified diagram of a computing environment in which oneor more embodiments may be practiced. In one embodiment, the computingenvironment is a cloud computing environment that includes a pluralityof physical computing devices (“physical machines”) networked togetherand managed by an entity such as a cloud controller. The diagram of FIG.1 illustrates components of the computing environment that monitor theperformance of various applications and physical machines deployedtherein. The monitoring components include a health monitoring server 10and a central monitoring server 20, one or both of which may beimplemented in a physical machine, in a virtual machine, or as softwareas a service. Central monitoring server 20 collects performance andchange data from a plurality of monitored machines 100, and healthmonitoring server 10 accesses such data and displays them to a systemadministrator, e.g., through a graphical user interface (GUI). Anexample of health monitoring server 10 is VMware vFabric Appinsight,which is commercially available from VMware, Inc. of Palo Alto, Calif.An example of central monitoring server 20 is VMware vFabric Hyperic,which is also commercially available from VMware, Inc. It should berecognized that, in alternative embodiments, the functions of healthmonitoring server 10 and central monitoring server 20 described hereinmay be carried out by a single server.

Each of health monitoring server 10, central monitoring server 20, andmonitored machines 100 includes hardware components of a conventionalcomputing device, such as processor and system memory, and system levelsoftware. Health monitoring server 10 further includes softwarecomponents running on top of its system level software, one of which isshown in FIG. 1 as health monitor 11. Health monitor 11 communicateswith central monitoring server 20 over a network to retrieve applicationperformance data and change data from central monitoring server 20through an application programming interface (API) 23. Health monitor 11also processes the retrieved performance data and change data togenerate various GUIs that are presented to the system administratorthrough a display 13.

Central monitoring server 20 further includes software componentsrunning on top of its system software, which are shown in FIG. 1 ascentral monitor 21 and API 23. Central monitor 21 communicates withmonitored machines 100 to collect application performance and changedata from them, and store them in an event store 30. Central monitor 21makes such collected data available to health monitor 11 through API 23.

Each of monitored machines 100 includes one or more applications 110running on top of its system software. It also includes a performancemonitoring agent 50 that monitors the performance and/or behavior ofapplications 110, such as central processing unit (CPU) usage, physicalmemory usage, and latency, and a change monitoring agent 60 thatmonitors changes to various files and folders designated by the systemadministrator as being monitoring targets. Some examples of applicationsthat can be tracked include web server applications such as Tomcat,JBoss, Websphere, Weblogic, and IIS, and database applications such asMySQL, PostgreSQL, Oracle, and MSSQL. Monitored machines 100 may bephysical or virtual machines. Physical machines include any computingdevice, such as server-grade computing platforms, desktop and laptopcomputing devices, and mobile computing devices, such as tabletcomputers and smart phones.

FIG. 2 is a conceptual diagram that illustrates various softwarecomponents of a monitored machine that enable tracking of changes madeto applications 110 running in the monitored machine. The presence ofapplications 110 running in monitored machine 100 is discovered throughapplication monitor 51, which keeps track of installed applications.Applications 110 running in monitored machine 100 may be designated forperformance monitoring and change tracking according to embodimentsdescribed herein, and each application that is designated for suchmonitoring and change tracking has a monitoring template 70 associatedtherewith. Monitoring templates 70 are stored and maintained in apersistent storage device of monitored machine 100, and for efficientaccess, may be cached in system memory of monitored machine 100. Asample monitoring template is illustrated in TABLE 1.

TABLE 1 <plugin package=“org.hyperic.hq.plugin.tomcat” name=“tomcat”version=“@project.version@”> <metrics name=“Some_metrics”>  </metrics> <server name=“Apache Tomcat” version=“5.5”  platforms=“Unix,Win32”> <property name=“VERSION_FILE” value=“server/lib/catalina- storeconfig.jar”/>  <plugin type=“config_track” class=“org.hyperic.hq.product.FileChangeTrackPlugin”>   <monitored>  <folder path=“conf” recursive=“true”  filter=“.*.properties|.*.xml|.*.policy”/>   <folder path=“bin”recursive=“false” filter=“.*.bat|.*.xml|.*.sh”/>   <folder path=“lib”recursive=“false” filter=“.*.jar”/>   <folder path=“webapps”recursive=“true”   filter=“.*\.jar|.*\.dll|.*\.class|.*\.jsp|.*\.php|.*\.pl|.-   *\.js|.*\.py|.*\.pyc|.*\.cgi”/>   <folder path=“data”recursive=“false” filter=“.*\.conf”/>   <!-- system variable PGDATA -->  <folder path=“%PGDATA%” recursive=“false” filter=“.*\.conf”/>  </monitored>  </plugin>  </server> </plugin>

Monitoring template 70 for an application identifies the applicationbeing monitored, and paths of folders and files to be monitored, somerecursively, for that application. In the example given in TABLE 1, theapplication being monitored is “Apache Tomcat” and the folders and filesthat are to be monitored recursively have a recursive flag set to betrue. Within a monitored folder, a regular expression filter is used todetermine which files are monitored. In addition, monitoring template 70may designate folders and files to be monitored using environmental orsystem variables so that files that affect the performance of theapplication and whose location on the file system is determined byenvironment/system variables may be monitored and tracked according toembodiments described herein. Using monitoring template 70, the foldersand files for any application being monitored may be custom defined. Byusing monitoring template 70, the number of folders and files to bemonitored can be reduced to a manageable (practical) number.

Change monitoring agent 60 monitors changes to various files and foldersof particular applications 110 being monitored as specified inmonitoring templates 70. As shown, change monitoring agent 60 includestwo modules. The first is an event processor 62 that receivesnotifications from the system software of monitored machine 100 that acertain folder or file has been changed. The particular components ofsystem software that are providing the notifications are a file system(FS) monitor 63, a database (DB) monitor 64, and for Microsoft Windows®systems, a registry monitor 65. In one embodiment, a Java® library knownas “jpathwatch” is used to monitor file system changes such as filecreation and deletion, file modification, file renaming, and changes insubfolders. As will be further described below in conjunction with FIG.3, event processor 62 evaluates the changes detected by FS monitor 63,DB monitor 64, and registry monitor 65 and produces change events forfurther processing by change tracker 61, which is the second modulewithin change monitoring agent 60 shown in FIG. 2. Change tracker 61receives change events from event processor 62 and processes theversioning of the changes to files using a versioning tool to store anychanges to the files in change repository 80, which may be provisionedin system memory or local storage. In one embodiment, the versioningtool known as Git is used.

FIG. 3 is a flow diagram of a method for generating change events when achange to a folder or file is detected. In the embodiment describedherein, this method is carried out by change monitoring agent 60, inparticular, event processor 62.

The method begins at step 310 when FS monitor 63 detects one or morechange events on the same file or folder during a configurable timeperiod, Tmin. In some situations, multiple change events relating toeffectively the same change are triggered, and this time period is setso that such change events can be captured and analyzed collectively. Inone embodiment, this time period is set as 2 milliseconds. At step 312,event processor 62 determines whether or not the file or folder shouldbe monitored, based on the installed applications that it discoveredthrough application monitor 51 and the monitoring templates 70. For eachmonitored folder, event processor 62 registers with the underlyingsystem software for change event notifications. If the file or folderfor which the change event is detected at step 310 is not specified in amonitoring template's filter of an installed application, the methodends. On the other hand, if the file or folder for which the changeevent is detected at step 310 is specified in a monitoring template ofan installed application, step 314 is executed where event processor 62determines the application ID (App ID) of the installed application forwhich the file or folder is being monitored. Then, at step 316, eventprocessor 62 determines whether a single file event is being processed.If a single event is being processed, step 318 is executed, where eventprocessor 62 issues a change event, including the App ID and the path tothe changed file to change tracker 61. If, on the other hand, a folderevent or multiple file events are being processed, step 320 is executed.

At step 320, event processor 62 determines whether it can transform afolder event or multiple file events into a single, normalized event,without relying on any OS-specific knowledge. For example, if two ormore modify events are triggered for the same file within the timeperiod, Tmin, event processor 62 may interpret all such modify events asa single modify event. In such situations, event processor 62, at step322, transforms the multiple events into a single, normalized event.Thereafter, step 330 is executed, where event processor 62 issues achange event, including the App ID and the path(s) to any changed filesto change tracker 61. The method ends after step 330 as there are nomore change events to process.

If, at step 320, event processor 62 determines that it cannot transforma folder event or multiple events into a single, normalized event,without relying on any OS-specific knowledge, steps 324 and 326 areexecuted. At step 324, event processor 62 determines the OS-type and, atstep 326, event processor 62 determines whether it can transform thefolder event or multiple events into a single, normalized event, withreliance on OS-specific knowledge. For example, in Windows® OS, a filecreation triggers five separate events—create, delete, recreate, modify,and modify. Event processor 62, at step 328, transforms all such eventsinto a single, normalized file creation event and, at step 330, issues achange event, including the App ID and the path to the newly createdfile, to change tracker 61. Another example is a folder copy event inWindows® OS, which triggers a folder create event and multiple fileevents for each file in the folder. Event processor 62, at step 328,transforms the folder copy event and the subsequent file events into asingle, normalized folder event and, at step 330, issues a change event,including the App ID and the paths to the files in the folder, to changetracker 61. With Linux® OS, where a folder copy event triggers only asingle folder event, event processor 62, at step 328, transforms thefolder copy event into a single, normalized folder event and, at step330, issues a change event, including the App ID and the paths to thefiles in the folder, to change tracker 61. In all three examples givenabove, the method ends after step 330 as there are no more change eventsto process.

In some situations where normalization may not be desired or cannot beimplemented, each change event is processed as a separate change eventat step 330. The method ends when it is determined at step 332 that allchange events have been processed.

FIG. 4 is a flow diagram of a method for versioning changes made tofiles of monitored applications. In the embodiment illustrated herein,change tracker 61 is performing the steps of this method and changetracker 61 is employing a versioning tool known as Git for steps 414through 424 of this method.

When change tracker 61 pulls a change event with a file path and achange type (create, delete or modify) from a queue of change eventsmaintained by event processor 62, change tracker 61 accesses the filesat the indicated file path. This method begins at step 410 with theaccessing of one such file. In some situations, the accessed file may bean archive file such as a JAR file or a WAR file, and change tracker 61checks for this at step 412. If the accessed file is an archive file,change tracker extracts a file to process from the archive file at step413, and step 414 is executed thereafter. If not, the method proceedsdirectly to step 414 and the accessed file is processed.

At step 414, the processed file is checked for changes. If there are nochanges, the file is not versioned and the method jumps to step 426. Ifthere are changes, a change ID and a time stamp for the changes isgenerated at step 416. If the changed file is binary file, as determinedat step 418, a hash of the binary file is generated (step 420) andstored in change repository 80 along with the change ID and the timestamp (step 424). If, on the other hand, the changed file is a textfile, a diff operation is performed on the text file against the priorversion of the text file to generate a diff file (step 422) and the difffile is stored in change repository 80 along with the change ID and thetime stamp (step 424).

If the file being processed is an archive file, as determined at step426, step 428 is executed to see if there is any more file to beextracted from the archive file. If there is any more file to beextracted from the archive file, the method returns to step 413 whereanother file is extracted. If there are no more files to be extractedfrom the archive file or the file being processed is not an archivefile, change tracker 61 executes step 430 to see if there are any morefiles to process. The method ends if there are no more files to process,and returns to step 410 if there is at least one more file to process.

As previously described, central monitor 21 of central monitoring server20 communicates with monitored machines 100 to collect performance andchange data from them, and store them in event store 30. The frequencyof the collection is configurable. The performance data that arecollected include App ID and time history of the CPU usage, memoryusage, and latency. The change data that are collected and stored inevent store 30 include the change ID, App ID, time stamp, path ofchanged file, and the diff file.

Health monitor 11 of health monitoring server 10 accesses theperformance data and the change data through API 23 to carry out amethod for generating, and processing inputs made to, an applicationperformance monitoring UI. The steps of this method are illustrated inFIG. 5.

At step 510, health monitor 11 displays a list of applications beingmonitored on a UI. A system administrator viewing the UI may select anapplication of interest, e.g., travel web server. Upon receiving thisselection at step 512, health monitor displays a performance monitoringUI for the selected application at step 514. A sample performancemonitoring UI is illustrated in FIG. 6A, and shows a time history of CPUusage 601 and latency 602. A user-selectable warning symbol 610 isdisplayed at a particular point in time of the illustrated timehistories as an indicator of when performance degradation of theselected application has occurred. When the system administrator makesan input selection (e.g., a mouse click or a tap on a touch screen) onwarning symbol 610 and health monitor 11 receives this selection at step516, health monitor 11 queries central monitoring server 20 for changeevents that are close in time to the performance degradation. Inresponse to the query, central monitoring server 20 searches change dataof the selected application maintained in event store 30 and returns therequested data to health monitor 11. At step 520, health monitor 11displays the change events on another UI, one example of which isillustrated in FIG. 6B. Upon viewing the change events, the systemadministrator may implement fixes or roll back the changes that causedthe performance degradation by making an input selection on top of oneof the change events (step 522). In the sample illustration of FIG. 6B,the code change implemented at 13:40:06 can be rolled back bymanipulating cursor 620 to that location and making an input selectionon that highlighted line. In response to the input selection, healthmonitor 11 issues an undo command identifying the change event by itschange ID through API 23 (step 524). Central monitoring server 20 passesthis undo command to the appropriate monitored machine and, in turn, themonitored machine executes the undo command using its versioning tool.

If the monitored machine is a virtual machine, as an alternative to thepinpoint rolling back, the entire virtual machine may be rolled back tothe most recent snapshot that was taken prior to the point in timeperformance degradation was observed.

The various embodiments described herein may employ variouscomputer-implemented operations involving data stored in computersystems. For example, these operations may require physical manipulationof physical quantities—usually, though not necessarily, these quantitiesmay take the form of electrical or magnetic signals, where they orrepresentations of them are capable of being stored, transferred,combined, compared, or otherwise manipulated. Further, suchmanipulations are often referred to in terms, such as producing,identifying, determining, or comparing. Any operations described hereinthat form part of one or more embodiments of the invention may be usefulmachine operations. In addition, one or more embodiments of theinvention also relate to a device or an apparatus for performing theseoperations. The apparatus may be specially constructed for specificrequired purposes, or it may be a general purpose computer selectivelyactivated or configured by a computer program stored in the computer. Inparticular, various general purpose machines may be used with computerprograms written in accordance with the teachings herein, or it may bemore convenient to construct a more specialized apparatus to perform therequired operations.

The various embodiments described herein may be practiced with othercomputer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, and the like.

One or more embodiments of the present invention may be implemented asone or more computer programs or as one or more computer program modulesembodied in one or more computer readable media. The term computerreadable medium refers to any data storage device that can store datawhich can thereafter be input to a computer system—computer readablemedia may be based on any existing or subsequently developed technologyfor embodying computer programs in a manner that enables them to be readby a computer. Examples of a computer readable medium include a harddrive, network attached storage (NAS), read-only memory, random-accessmemory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, aCD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, andother optical and non-optical data storage devices. The computerreadable medium can also be distributed over a network coupled computersystem so that the computer readable code is stored and executed in adistributed fashion.

Although one or more embodiments of the present invention have beendescribed in some detail for clarity of understanding, it will beapparent that certain changes and modifications may be made within thescope of the claims. Accordingly, the described embodiments are to beconsidered as illustrative and not restrictive, and the scope of theclaims is not to be limited to details given herein, but may be modifiedwithin the scope and equivalents of the claims. In the claims, elementsand/or steps do not imply any particular order of operation, unlessexplicitly stated in the claims.

Virtualization systems in accordance with the various embodiments may beimplemented as hosted embodiments, non-hosted embodiments or asembodiments that tend to blur distinctions between the two, are allenvisioned. Furthermore, various virtualization operations may be whollyor partially implemented in hardware. For example, a hardwareimplementation may employ a look-up table for modification of storageaccess requests to secure non-disk data.

Many variations, modifications, additions, and improvements arepossible, regardless the degree of virtualization. The virtualizationsoftware can therefore include components of a host, console, or guestoperating system that performs virtualization functions. Pluralinstances may be provided for components, operations or structuresdescribed herein as a single instance. Finally, boundaries betweenvarious components, operations and data stores are somewhat arbitrary,and particular operations are illustrated in the context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within the scope of the invention(s). Ingeneral, structures and functionality presented as separate componentsin exemplary configurations may be implemented as a combined structureor component. Similarly, structures and functionality presented as asingle component may be implemented as separate components. These andother variations, modifications, additions, and improvements may fallwithin the scope of the appended claim(s).

We claim:
 1. A method for tracking changes to data containers includingone or more first data containers corresponding to a first applicationand one or more second data containers corresponding to a secondapplication, said method comprising: identifying one or more of thefirst data containers to be monitored for configuration changes of thefirst application based on a first expression that is stored inassociation with the first application and identifies the one or more ofthe first data containers; identifying one or more of the second datacontainers to be monitored for configuration changes of the secondapplication based on a second expression that is stored in associationwith the second application and identifies the one or more of the seconddata containers; detecting one or more changes to a monitored datacontainer; and storing data representative of a changed version of themonitored data container responsive to detecting the one or more changesto the monitored data container.
 2. The method of claim 1, wherein thefirst and second data containers include at least one of files andfolders, and wherein the first and second expressions specify paths ofthe at least one of files and folders that are to be monitored.
 3. Themethod of claim 2, wherein the first and second expressions specifyenvironmental variables, the values of which define paths of the atleast one of files and folders that are to be monitored.
 4. The methodof claim 2, wherein detecting the one or more changes to the monitoreddata container includes detecting multiple file events, and wherein themethod further comprises: transforming the multiple file events into asingle, normalized file event.
 5. The method of claim 1, furthercomprising: performing a diff operation on the changed version of themonitored data container against a previous version of the monitoreddata container and storing results of the diff operation as the datarepresentative of the changed version of the text file.
 6. The method ofclaim 1, wherein the one or more changes to the monitored data containerare detected within a predetermined time period.
 7. The method of claim1, wherein the first and second applications are deployed on avirtualization system.
 8. The method of claim 1, further comprising:collecting and storing performance data associated with the first andsecond applications.
 9. The method of claim 8, wherein the collected andstored performance data include at least one of CPU usage, memory usage,and latency.
 10. The method of claim 8, further comprising: displayingan indication of the stored performance data associated with the firstor the second application along with an indication of stored datarepresentative of changed versions of the first or the secondapplication.
 11. A non-transitory computer-readable storage mediumstoring instructions, which when executed by a computer system, performoperations for tracking changes to data containers including one or morefirst data containers corresponding to a first application and one ormore second data containers corresponding to a second application, theoperations comprising: identifying one or more of the first datacontainers to be monitored for configuration changes of the firstapplication based on a first expression that is stored in associationwith the first application and identifies the one or more of the firstdata containers; identifying one or more of the second data containersto be monitored for configuration changes of the second applicationbased on a second expression that is stored in association with thesecond application and identifies the one or more of the second datacontainers; detecting one or more changes to a monitored data container;and storing data representative of a changed version of the monitoreddata container responsive to detecting the one or more changes to themonitored data container.
 12. The non-transitory computer-readablestorage medium of claim 11, wherein the first and second data containersinclude at least one of files and folders, and wherein the first andsecond expressions specify paths of the at least one of files andfolders that are to be monitored.
 13. The non-transitorycomputer-readable storage medium of claim 12, wherein the first andsecond expressions specify environmental variables, the values of whichdefine paths of the at least one of files and folders that are to bemonitored.
 14. The non-transitory computer-readable storage medium ofclaim 12, wherein detecting the one or more changes to the monitoreddata container includes detecting multiple file events, and wherein theoperations further comprise: transforming the multiple file events intoa single, normalized file event.
 15. The non-transitorycomputer-readable storage medium of claim 11, the operations furthercomprising: performing a diff operation on the changed version of themonitored data container against a previous version of the monitoreddata container and storing results of the diff operation as the datarepresentative of the changed version of the text file.
 16. Thenon-transitory computer-readable storage medium of claim 11, wherein theone or more changes to the monitored data container are detected withina predetermined time period.
 17. The non-transitory computer-readablestorage medium of claim 11, wherein the first and second applicationsare deployed on a virtualization system.
 18. The non-transitorycomputer-readable storage medium of claim 11, the operations furthercomprising: collecting and storing performance data associated with thefirst and second applications.
 19. The non-transitory computer-readablestorage medium of claim 18, the operations further comprising:displaying an indication of the stored performance data associated withthe first or the second application along with an indication of storeddata representative of changed versions of the first or the secondapplication.
 20. A system, comprising: a processor; and a memory,wherein the memory includes an application program configured to performoperations for tracking changes to data containers including one or morefirst data containers corresponding to a first application and one ormore second data containers corresponding to a second application, theoperations comprising: identifying one or more of the first datacontainers to be monitored for configuration changes of the firstapplication based on a first expression that is stored in associationwith the first application and identifies the one or more of the firstdata containers, identifying one or more of the second data containersto be monitored for configuration changes of the second applicationbased on a second expression that is stored in association with thesecond application and identifies the one or more of the second datacontainers, detecting one or more changes to a monitored data container,and storing data representative of a changed version of the monitoreddata container responsive to detecting the one or more changes to themonitored data container.